emarks 

Many X-ray systems have been proposed. For example, U.S. Pat. No. 5,3 19,547 and 
U.S. Pat. No. 5,838,758, issued to Krug et al, describe an X-ray line scanner for 
automatically scanning luggage, U.S. Pat. No. 5,367,552, issued to Peschmann, and U.S. 
Pat. No, 6,185,272, issued to Hiraoglu, describe a CT scanner for automatically scanning 
luggage. However, all the previous patents disclose new and/or stand alone systems 
and/or devices while the present invention discloses a system to enhance/upgrade 
regular screening devices with pattern and/or object recognition capabilities. 

Building a new x-ray screening device requires the production and the assembly of 
various parts (including x-ray tubes, conveyer belt, monitor, etc) that already exist in a 
regular screening device. 

On the other hand, enhancing/upgrading a regular screening device using the system 
disclosed in the present invention can be accomplished with commercially available, 
mass-produced parts and devices comprising a personal computer system, a computer 
monitor, and a video capture card. 

Therefore building new x-ray screening devices, as disclosed in all the previous 
patents, costs a lot more than enhancing/upgrading regular screening devices (see also 
correction#10,#ll). 

In plain English, all the patents issued so far are for new, stand-alone, expensive 
automatic screening devices while the present invention is an inexpensive enhancement 
for the existing screening devices. It doesn't automatically single out suspicious luggage, 
but it marks all suspicious items in the luggage being examined by the screening device's 
operator so that the operator can take a look at them more carefully (see correction #11). 

I listed all the corrections I made in a separate document (Correction List). I also 
attached a clean (with no remarks) version of the amended patent application and the 
corrected marked version (with remarks). 
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Cross-Reference to Related Applications 

Not applicable. 



Background of the Invention 

This invention relates generally to image and document image understanding, and 
more particularly , to a system that can detect or recognize certain objects in a screening 
process. 

Screening for hazardous objects using a screener device is a very demanding task that 
requires both accuracy and efficiency. Human factors comprising sleepiness, fatigue, 
boredom, and inadequate training may affect the ability of a person to do this task accurately 
and efficiently. Unfortunately, this kind of failures may potentially lead to a disaster. 




Different X-rav systems have been proposed. 



For example, U.S. Pat. No. 5,319.547 and U.S. Pat. No. 5,838,758. issued to Krug et ah 
describe an X-ray line scanner for automatically scanning luggage, U.S. Pat. No. 5,367,552. 
issued to Peschmann, and U.S. Pat. No. 6,185,272, issued to Hiraoelu, describe a CT scanner 
for automatically scanning luggage. Additionally, manual (human-operated) screening 
devices have been around for many years. 

While all of these devices are designed to automatically detect explosives, they are 
stand-alone devices that cost a lot of money and may not be as reliable as they should be. 
Therefore, it is virtually impossible to replace all regular screening devices with their 
automatic counterparts. 



present invention is to enhance regular screening devices with object recognition capabilities. 
To minimize the cost of such enhancement, the system uses TWAIN, a widely accepted data 
acquisition protocol. As a result the enhancement can be accomplished with commercially 
available computer systems such as PC (personal computer with AMD or Intel-based 
microprocessor) and/or Mac. 



Summary of the Invention 



The primary object of the invention is to recognize potentially hazardous objects 
during a screening process. 

Another object of the invention is to minimize screener's failure to recognize or to 
detect potentially hazardous objects during a screening process by recognizing and marking 
said objects automatically when they are displayed on a monitor. 

The system and method of this invention recognize objects trained by the user. Said 
system categorizes said objects into several classes, and marks said objects according to their 
classes. The system displays the representation of the recognized objects hierarchically. Each 
parent node displays a class of objects. Said user may expand said parent node to display the 
representation of said recognized objects that belong to that class. Once displayed, said user 
may choose the representation of an object to pinpoint the location and the class of said 
object. 

The system comprises an image processing subsystem, a recognition subsystem, and a 
training subsystem. 

The image processing subsystem acquires an image from a screening or image 
acquisition device such as an x-ray screening device by using standard TWAIN protocol. For 
a device without any compatible interfaces, a special adaptor that convert the available 
interface to a supported interfaces such as universal serial bus or parallel port along with an 
appropriate driver can be used. The image acquired from the device is processed ftirther to 
increase the performance of the system. 

The object recognition subsystem uses the information acquired and processed by the 
image processing subsystem about the objects and their locations. The object recognition 
subsystem determines the boundary of each object in the image and recognizes them by using 
a pattern recognition engine tolerant to rotation and size. The object recognition subsystem 
recognizes each object in the image and categorizes each recognized object into object 
classes. 

The training subsystem is used to teach the object recognition subsystem to recognize 
new kinds of objects and re-leam old objects. 



Brief Description of the Drawings 



. The foregoing features and other aspects of this invention will now be described in 
accordance with the drawings in which: 

Figure.l is a diagram of the suggested application and requirement or configuration of the 
system to be used with a screening device. 

Figure.2 is a UML diagram of key elements in the system. 

Figure.3 is a diagram of the neural networks used to recognize pattern in the object 
recognition engine in the system. 

Detailed Descriptions of the Preferred Embodiments 

Detailed descriptions of the preferred embodiment are provided herein. It is to be 
understood, however, that the present invention may be embodied in various forms. 
Therefore, specific details disclosed herein are not to be interpreted as limiting, but rather as a 
basis for the claims and as a representative basis for teaching one skilled in the art to employ 
the present invention in virtually any appropriately detailed system, structure, or manner. 

Referring now to Figure.l, the system is shown to comprise a screening device 1 . Said 
screening device 1 comprises a generic x-ray screening device. 

The system is shown to further comprise an adaptor 2. Said adaptor 2 converts video 
signal output fi"om said screening device 1 to digital format. Said digital format follows 
standard and port that can be recognized by the system. 

The system is shown to fiirther comprise a computer system 3. The computer system 
3 comprises personal computer that can run the software part of the system. Said computer 
system 3 displays data firom said screening device 1 and pinpoints objects said computer 
system 3 recognizes as hazardous objects. 



An operator 4 operates the system. Said operator 4 performs more thorough checking 
whenever the system detects possible hazardous objects. 

Referring now to Figure.2, the UML diagram of the system is shown to comprise 
TWAIN interface 20. Said TWAIN interface may control data acquisition from any TWAIN- 
compatible image acquisition device comprising a screening device 10. Said TWAIN 
interface then produces an image 30 of the actual objects being screened. 

The system is shown to further comprise an image-processing subsystem 40, which 
comprises an image processing engine 41 and an object recognition engine 42. 

Said image-processing engine 41 receives said image 30 and applies image- 
processing techniques to enhance the quality of said image 30. Said image-processing 
techniques comprise dilation, image-depth conversion, and gray scaling. Said image- 
processing engine 41 converts said image 30 into several two-dimensional array image 
matrixes 43. Each image matrix 43 comprises a filtered version of said image. 

The object-segmentation engine 42 uses image matrix 43 to get the boundary of each 
object. The object-segmentation engine stores the information about said boundary of each 
object in a list of objects 44. 

The system is shown to further comprise a recognition subsystem 50, which 
comprises an object recognition engine 51. 

The object recognition engine 51 receives said image matrix 43 and said list of objects 
44. The object recognition engine 51 retrieves the representation of each object in said image 
matrix 43 using data from said list of objects 44. The object recognition engine 51 produces 
object info 53 comprising the class and the hazard level of each object using a priority list 52. 
Said priority list 52 comprises a list of all classes of objects and their hazard levels. The 
object recognition engine 5 1 uses pattern recognition engine 54. Said pattern recognition 
engine 54 is a neural network pattern recognition engine tolerant to rotation and scaling. 



The system is shown to fiirther comprise a user interface/object viewer 60. The user 
interface/object viewer 60 displays the class of each object recognized by said object 
recognition engine 51 hierarchically, grouped by their hazard levels. Said user 
interface/object viewer 60 pinpoints an associated object if a user chooses a class that 
represents that object. The way user interface/object viewer 60 pinpoints an object depends 
on the hazard level of that object. A monitor 70 displays the user interface/object viewer to 
said user. 

Referring now to Figure.3, the diagram of the artificial neural networks used to 
recognize pattern in the object recognition engine in the system shown to comprise input 
pattern 100. Said input pattern 100 is the pattern that will be recognized by the neural 
networks. Each pattern is a representation of an object the recognition system is trying to 
recognize. 

The neural network is shown to further comprise feature templates layer 1 10. Feature 
templates 1 10 are used to extract certain features from said input pattern 100. Feature 
templates 1 10 are arranged in several clusters, each cluster has the same number of templates. 

The neural network is shown to fijrther comprise input neurons 120. Said input 
neurons 120 form an input layer. Each neuron in said input neurons 120 receives input fit)m 
the result of feature extraction by a template in said feature templates 1 10 layer. Said input 
neurons are arranged in several clusters, each cluster has the same number of neurons. The 
number of neurons in each cluster is equivalent to the number of templates in a cluster in said 
feature templates 1 1 0. 

The neural network is shown to further comprise shift registers or ring buffers 130. 
Each shift register contains a certain number of elements. Each element receives input from a 
neuron in said input layer 120. The number of elements in each shift register is equivalent to 
the number of neurons in a cluster in said input layer 120. 

The neural network is shown to further comprise output neurons 140. Said output 
neurons 140 form an output layer. Many kinds of neural networks can be used in this layer, 
comprising variants of multiplayer perceptrons (MLP) and variants of radial basis function 
(RBF) networks. This output layer receives input from said shift registers 130. 



While the invention has been described m connection with a preferred embodiment, it 
is not intended to limit the scope of the invention to the particular form set forth. On the 
contrary, it is intended to cover such alternatives, modifications, and equivalents as may be 
included within the spirit and scope of the invention as defined by the appended claims. 



Claims 



that receives data from 



What is claimed is: 

1 . A system md method= 

aoquioition device comprising a screening device, tries to recognize each objoot all 
objects in said data, and pinpoints each object all objects ft said system and 
method is trained to recognize along with said object's class and hazard level. 




5. The method of claim 1 further comprises 

ksing an audio output comprising a horn or a sound generator . 



6. A computer program product 

onmjlutGr program logic recorded thereon that comprising a computer useable 
medium having computer program logic recorded on it for enabling a computer 

system to : 

a. receive data from an image acquisition device comprising a regular an x- 
ray screening device 

b. try to recognize each object in said data 

c. pinpoint each object it is trained to recognize along with its class and 
hazard level. 



7. The computer program of claim 6 wherein said program further comprises a 
remote databaoo and/or shared data of list of objects . 
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Abstract of the Disclosure 




oorooning dcvioc with a vidoo to USB adapton The data from oaid dovjoo io pro proooaood to 




convoroionit and gray ooaling. After tho onhanoomont procooo» information about oaoh objoot 




Electronic Object Detectibn is a system and method that enhances a screening device 
by recognizing objects in images or data acquired from said screening device and mark said 
objects if they can be hazardous. Said system tries to recognize each object using an object 
recognition engine tolerant to size and rotation and displays the information about the 
possible class of each object and its hazard level. 
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(High Level Abstraction) 



Screening Device 10 



50. Recognition 
Subsystem 



54. Pattern 
Recognition 



30. Image 













' <Acquir« 



20.TWAIN 
Interfiace 



40. Image Processing 
Subsystem 



41 . Image 
Processing 



42. Object 
Segmentation 



T 



<Uses 



i i_ 



43. Image Matrix 



<Uses 



ReoognizB> 



44. List of Objects 
s 



51. Object 
Recognition 



53. Object Info 



Ums> 



52. Priority List 



60. User 
Interface/ Object 
Viewer 



Ul / Object Viewer 



<Ums 



70. Monitor 



□ 



Figure.2 



3/3 




Figure.3 
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U.S. Patent Application of: 
Yandi Ongkojoyo 



Correction List: 

1. Claim 6: 

a. Sub (a): changed "receives" to "receive" to accommodate correction #6 
sub (d) 

2. Claims 2,3,4,5: 

a. Claim 2: Cancelled 

b. Claim 3: Cancelled 

c. Claim 4: Cancelled 

d. Claim 5: Changed "The method of claim 1 further comprises other Idnd 
of uocr intcrfacco, comprioing audio output" to "The method of claim 1 
further comprises an audio output comprising a horn or a sound generator 

Unclear, 

3. Claims 2, 3, 5, 7, 8: 

a. Claim 2: Does not apply. Claim cancelled. See correction #2 sub (a). 

b. Claim 3: Does not apply. Claim cancelled. See correction #2 sub (b). 

c. Claim 5: Does not apply. Claim corrected. See correction #2 sub (d). 

d. Claim 7: Changed: "The computer program of claim 6 wherein said 
program further comprises a remote databaac " to "The computer program 
of claim 6 vs^herein said program further comprises a remote and/or shared 
data of list of objects " 

e. Claim 8: Cancelled. 

4. New abstract submitted, 

5. Notified. 

6. Claims 1-11: 

a. Claim 1 : Changed "A system and method 
rcooivoo data from 

a screening device, tries to recognize each object in said data, and 
pinpoints each object it is trained to recognize along wdth 4ts class and 
hazard level." to "A system ^ method that receives data from a 
screening device, tries to recognize all objects in said data, and pinpoints 
all objects said svstem and method is trained to recognize along with said 
object's class and hazard level." -> Requested, ambiguous, 

b. Claim 1 : Doesn't apply. Claim Corrected. See correction #6 sub (a). 

c. Claim 2, 3, and 4: Don't apply. Claims cancelled. 

d. Claim 6: Changed "A computer program product 



.99 



to "A computer program product comprising a computer useable medium 
having computer program logic recorded on it for enabling a computer 
system to:" -> Requested 
Claim 9: Cancelled 
Claun 10: Cancelled 
Claim 1 1 : Cancelled 

Claim 2, 3, 4, 5: Cancelled. See correction #2. 
Claim 7: Changed. See correction #3 sub (d). 

7. Notified. 

8. Claim 9: Cancelled. See correction #6 sub (e). 



e. 

f. 

g- 
h. 

« 

1. 



9. Claim 9: Cancelled. See correction #6 sub (e). 

10. Claims 1,6, 7: 

a. Hiraoglu (U.S Patent 6,185,272) discloses a system equipped vsdth a CT 
scanner to recognize items, while the present invention discloses a system 
to enhance/upgrade regular screening devices with pattern and/or 
object recognition capabilities. Building a new x-ray screening device 
costs a lot more than enhancing/upgrading a regular screening device. 

11. Prior art (1-4): 

a. Changed paragraph 3, Background of Invention from: " Upgrading the 



ocrccncr dcvi 



wcvcr, It IS an 




'Different X-ray systems have been proposed. For example, U.S. Pat. No. 
5,319,547 and U.S. Pat. No. 5,838,758, issued to Krug et al, describe an 
X-ray line scanner for automatically scanning luggage, U.S. Pat. No. 
5,367,552, issued to Peschmann, and U.S. Pat. No. 6,185,272, issued to 
Hiraoglu,, describe a CT scanner for automatically scanning luggage. 
Additionally, manual (human-operated) screening devices have been 
around for many years." Added references 

b. Added a hew paragraph after paragraph 3: "While all of these devices are 
designed to automatically detect explosives, they are stand-alone devices 
that cost a lot of money and ihay not be as reliable as they should be. 
Therefore, it is virtually impossible to replace all regular screening devices 
with their automatic counterparts." To clarify in plain English 

c. Changed paragraph 4 from: "* 




\ ^ the oporatdr of the ocrooriing 
ari bd intbrfacod mth any tViMN - 
tt wth a 01 




iced \\ith the sorociling dcviooo aire 
"The object of the present invention is to enhance regular screening 
devices with object recognition capabilities. To minimize the cost of such 
enhancement, the system uses TWAIN, a widely accepted data acquisition 
protocol. As a result, the enhancenient can be accomplished with 
commercially available computer systems such as PC (personal computer 
with AMD or Intel-based microprocessor) and/or Mac." -> Unclear 
d. Ill plain Eiiglish, all the patents issued so far are for new, stand-alone, 
expensive automatic screening devices while the present invention is an 
inexpensive enhancement for the existing screening devices. It doesn't 
automatically single out suspicious luggage, but it marks all suspicious 
items in the luggage being examined by the screening device's operator so 
that the operator can take a look at them more carefully. 
12. Notified. 



13. Claim 1 1 : Cancelled See correction #6 sub (g). 

14. Claim 10: Cancelled See correction #6 sub (f). 

15. Claim 10: Cancelled See correction #6 sub (f). 

16. Notified. 
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Abstract of the Disclosure 



Electronic Object Detection is a system and method that enhances a screening device 
by recognizing objects in images or data acquired from said screening device and mark said 
objects if they can be hazardous. Said system tries to recognize each object using an object 
recognition engine tolerant to size and rotation and displays the information about the 
possible class of each object and its hazard level. 



